A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce

Abstract Large-scale distributed training mainly consists of sub-model parallel and parameter synchronization. With the expansion workers, efficiency synchronization will be affected. To tackle this problem, we first propose 2D-TGA, a g rouping A llReduce method based on two-dimensional t orus topology. This synchronizes model parameters by grouping makes full use bandwidth. Secondly, algorithm, 2D-TGA-ADMM, which combines 2D-TGA with alternating direction multipliers (ADMM). It focuses reduces wait time among workers in process. Finally, experimental results Tianhe-2 supercomputing platform show that compared $${\mathtt {MPI\_Allreduce}}$$ MPI _ Allreduce , could shorten $$33\%$$ 33 % .